The increase in cancer cases, the democratization of healthcare, even the recent pandemic, are some of the numerous reasons pointing out that there is a great need in leveraging AI technology in patients' care practice. However, while AI has demonstrated the capability to be a valuable companion to practitioners, with respect to meeting accuracy levels, removing bias and increasing diagnostic throughput, its adoption to clinical practice is still slow. While the problem is more complex, in this instructional workshop we will focus on two critical aspects of adoption. AI technologies, notably deep learning techniques, may hide inherent risks such as the codification of biases, the weak accountability and the bare transparency of their decision-making process. AI technology needs to both improve the diagnostic power of the data processed but also provide evidence for the prediction in a user understandable way.